Image transforming vision enhancement device

ABSTRACT

Image transforming vision enhancement device that enhances vision by transforming images provided by one or more cameras into modified images projected on one or more displays. The system may be embedded in glasses, contact lenses, binoculars, or other vision devices, in computer screens, or in components of moving vehicles. Image transformations may include modifying colors to assist colorblind users or to highlight color ranges, mapping invisible frequencies into visible colors, adding labels or graphics, and generating time-varying images with flashing or changing features. Images from multiple cameras may be combined, providing users with panoramic vision from a single device. Low light vision may be enhanced, and excessive glare may be attenuated. The system may magnify images with a variable magnification. User interfaces may be provided to configure and customize the image transformations.

This utility patent application is a continuation of U.S. Utility patentapplication Ser. No. 14/804,197, filed 20 Jul. 2015, which claims thebenefit of U.S. Provisional Patent Application No. 62/122,130, filed 14Oct. 2014, the specifications of which is hereby incorporated herein byreference.

BACKGROUND OF THE INVENTION

Field of the Invention

This invention relates generally to the field of correction,augmentation, and enhancement of vision. In particular, but not by wayof limitation, one or more embodiments of the invention relate to animage transforming vision enhancement device, for example that may beutilized to transform images using color transformations and mapping,magnification, cropping, rotation, scaling, selection, and overlay ofgraphics or labels and generating time-varying images with flashing orchanging features to indicate colors for example, or any combinationthereof.

Description of the Related Art

Vision enhancement devices such as specialized eyeglasses or contactlenses are known in the art. In particular, eyeglasses to assistcolorblind users are known. For example, in U.S. Pat. No. 4,300,819,Taylor teaches glasses with one clear and one colored lens, which assistcolorblind users in distinguishing between colors that they normallyfind difficult to distinguish. Variations on this invention haveincluded for example glasses with lenses of two different colors, astaught by Davis in U.S. Pat. No. 5,917,573, and glasses that can bemanufactured with lenses that provide various shapes of spectralfilters, as taught by Chen in US Patent Publication US20110090453.Contact lenses using the same principle of coloring one of the lensesare also known, as taught for example by Zeltzer in U.S. Pat. Nos.3,586,423 and 3,701,590. These devices use analog filters and colormasks to modify the light reaching a user's eye. They are thereforelimited to the specific color filtering provided by the individuallyconfigured lenses. Since analog filters can only remove or attenuatelight of particular wavelengths, these filters also typically darken theresulting image, which may often reduce visibility. Analog filters donot have the ability to add or amplify light, which would be useful insome situations where normal light is too low, such as for night vision.

Other types of vision enhancement devices include “night vision” gogglesand cameras, which use more specialized (often analog) technique torepresent a particular non-visible set of frequencies. Here, most normalcolor images are replaced by a nearly monochromatic image showing amapping of infrared light to a visible image.

More recently some computer and mobile phone based systems have emergedthat provide limited digital processing of images to assist colorblindusers. For example, Jones in US Patent Publication US20140153825 teachesa technique to add patterns to specified colors for display of images ona screen.

While the devices and techniques known in the art have proven helpful tosome users, they have not provided full color vision to colorblindusers. They have also been designed for very specific applications orcategories of users. No general-purpose digitally programmable visionenhancement device is known in the art that can provide a flexible,configurable system for altering and enhancing images. No known systemor method can alter or enhance images in multiple ways, including colortransformations, addition of graphics or captions, magnifying images, orgenerating images that vary over time. No known systems provide flexibletransformation and augmentation of images in the visible spectrum alongwith display of invisible frequencies such as infrared or ultraviolet.No known systems integrate such general and flexible imagetransformations into a vision enhancement device that can be embedded inglasses, contact lenses, binoculars, or other vision devices.

For at least the limitations described above there is a need for animage transforming vision enhancement device.

BRIEF SUMMARY OF THE INVENTION

One or more embodiments described in the specification are related to animage transforming vision enhancement device. Embodiments of theinvention may include or otherwise utilize one or more cameras, one ormore displays, and one or more processors. At least one embodimenttransforms images captured from the cameras and projects the transformedimages onto the displays. Some embodiments may associate displays withthe eyes of a user; for example there may be left display positionednear the user's left eye and a right display positioned near the user'sright eye. The image transformation may perform any desiredtransformation on the images, including without limitation colortransformation, for example mapping colors to different colors,magnification, cropping, rotation, scaling, selection, and overlay ofgraphics or labels and time-shifting techniques that for example varyintensity over time or flash or show patterns in particular areas havinga color or range of colors. Color transformations map values from one ormore camera color spaces to one or more display color spaces.Embodiments may use any convenient color spaces, including withoutlimitation RGB, RGBA, HSV, HSL, HSI, indexed color, or any other colorrepresentation.

One or more embodiments of the invention include displays that areconfigured to be placed or worn proximal to one or both of the eyes of auser. Such embodiments may for example be embedded in eyeglasses,sunglasses, contact lenses, monocles, visors, helmets, diving helmets,diving masks, welding helmets, goggles, protective eyewear, or othervision devices. In some embodiments, one or more cameras may beconfigured to aim in the direction of the user's normal sight and tocover all or part of the user's normal field of view. The cameras mayalso be embedded in the vision devices, or they may be external in someembodiments. For example, in some embodiments the cameras may be placedalong the front of the lenses of eyeglasses, and the backs of the lensesmay consist of displays that show images generated by the imagetransformation function executed in the system's processor. In someembodiments of the invention all of the components may be miniaturizedand embedded in normal eyewear, for example in glasses, contact lensesor intraocular lenses.

In one or more embodiments of the invention, some or all components ofthe system may be embedded in devices that are not typically worn by auser, but are viewed or viewable by one or more users. Such devices mayfor example include a television, a movie screen, a stadium monitor, akiosk, a game system, a desktop computer, a laptop computer, a tabletcomputer, a mobile phone, a personal digital assistant, a medicaldevice, a scientific device, or any device that a user may employ toview pictures, photos, graphics, media, videos, or other information.Devices may be viewed or viewable by a single user, or by a group ofusers, including potentially large groups of users such as a crowd at atheater or a sporting event. In one or more embodiments the display ordisplays of the system may be connected to a network, and the displaysmay receive images via the network. The network may be wired orwireless. The network may be public or private, and it may be a widearea network, a metropolitan area network, a local area network, or anynetwork that may deliver images from one location or device to adifferent location or different device. In one or more embodiments thenetwork may be the public Internet, and the delivery of images mayinclude for example display of images on a web page, or streaming ofvideo or other media over the Internet. Embodiments that supportInternet delivery of images may for example embed image transformationfunctions into web servers, proxy servers, browsers, Internet ServiceProvider gateways, or any other node or nodes in the transmission pathbetween the source of the image and the display.

One or more embodiments provide a zoom capability to magnify, or toreduce magnification, of the camera images. Magnification levels may beeither fixed or variable, and may or may not be under user control.Embodiments with magnification capabilities may be embedded for examplein binoculars, a monocular, a telescope, or a microscope. Magnificationcapabilities may also be provided in embodiments that are embedded inglasses or contact lenses or other traditional eyewear.

One or more embodiments combine images from two or more cameras onto asingle display image. For example, such an embodiment may provide apicture-in-picture display with the main image representing a user'sforward vision, and a smaller in-picture display coming from abackward-facing camera. Some embodiments may include cameras aiming inall directions and may combine images from these cameras into a completepanoramic display for the user. Embodiments with multiple cameras mayprovide cameras that aim in directions that the user cannot normallysee, such as backwards, sideways, or up. Embodiments may includenon-aligned cameras or other transformation techniques to correctheterotropia for example.

In some embodiments the image transformation functions may vary acrossdisplays. For example, a left eye display transformation may bedifferent from a right eye display transformation. Colors in a cameraimage may be mapped to different colors in a left display and a rightdisplay, for example. Such embodiments may for example support 3D visionusing anaglyph techniques that apply different color filters to astereoscopic image

Embodiments of the invention provide vision enhancement capabilities toassist colorblind users. Such users have difficulty distinguishingbetween certain colors. The image transformation functions of theseembodiments may map colors that are difficult or impossible for acolorblind user to distinguish into different colors that such a usercan easily distinguish. For example, the brightness of colors may beselectively altered to make color differences more apparent. As opposedto the typically used technique to correct colorblindness by providingan analog colored filter in one lens of a pair of glasses, which has theeffect of making overall images darker, one or more embodiments of theinvention may for example use the opposite technique of addingbrightness in selected wavelengths to assist with color differentiation.This technique of adding brightness is not possible in the analogfilters of the prior art because analog filters can only remove orattenuate light in selected wavelengths. Embodiments that add brightnessto selectively provide significant benefits over the inherentlysubtractive filters in the prior art, because the overall brightness ofan image may be preserved or enhanced.

In one or more embodiments, the image transformation function may modifya subset of the pixels of the camera images, where the desired subset isspecified by an image mask. Embodiments may define an image mask usingany desired pixel characteristics, including for example a range or aset of ranges of colors determining which pixels are included in theimage mask. Image masks defined by pixel characteristics are adaptivemasks that are determined for each image based on the contents of thatimage. Embodiments may also use fixed image masks that select subsets ofpixels based on fixed geometric boundaries. Embodiments may usecombinations of adaptive image masks and fixed image masks. Pixelsselected by an image mask or a combination of image masks may bemodified after selection in any desired manner. One or more embodimentstransform images to highlight pixels or regions with colors in aparticular range of interest. For example, selected colors may bebrightened to make them stand out more clearly. Other embodiments mayplace graphics or labels near or around regions with specified colors tomake them clearer to a user. Such embodiments may for example haveindustrial applications for inspections, where subtle color differencesmay signal important issues but where such subtle color differences maybe difficult to discern without image enhancement.

One or more embodiments of the invention modify the brightness ofselected pixels, in order to improve visibility of dark scenes orregions, or to reduce excessive brightness or glare of overly brightpixels or regions, or both. Such embodiments may assist with nighttimeor low-light vision. An example application is an embodiment thatdarkens colors above a certain level of brightness in order to reduceglare from headlights of oncoming or following cars. In some embodimentsthe image transformation could both darken excessively bright pixels andbrighten excessively dark pixels; this combined effect providesadditional advantages for nighttime driving where excessive glare andpoor visibility in darkness are both potential problems. Such anembodiment may for example be embedded in driving glasses, or into thewindshield or rear-view mirror or other component of a car or othermoving vehicle.

One or more embodiments provide capabilities that allow users to seelight frequencies outside the human visible frequency range. Cameras inthese embodiments may capture frequencies such as infrared orultraviolet, or any other electromagnetic frequency such as gamma rays,x-rays, microwave, or radio waves. Any sensor that can capture any rangeor any set of ranges of the electromagnetic spectrum may be used as acamera in embodiments of the invention. In embodiments that capturenon-visible frequencies, the image transformation functions of theseembodiments may map these invisible frequencies into visible frequenciesfor display. Illustrative applications for such embodiments may includefor example visualizing heat losses, monitoring radiation levels forsafety, searching for hidden transmitters and hidden listening orrecording devices, and configuring and optimizing wireless networks.Various techniques may be used to map invisible frequencies into thevisible frequency range. In some embodiments, the image transformationfunction partitions the display color space into two subspaces: onesubspace is used to represent visible frequencies, and the other is usedto represent invisible frequencies. For example, if the display colorspace is RGB, then the red component may be used to represent invisible(such as infrared) frequencies, and the green and blue components may beused to represent visible frequencies. Some embodiments simply discardcertain visible frequency information and use the remaining visiblespectrum to represent invisible frequencies. Some embodiments insteadcompress the visible frequency range rather than discarding portions ofit. For example, in one or more embodiments the camera color space anddisplay color space may be HSV (hue-saturation-value) or some similarcolor space. In these embodiments the hues of visible frequencies aretypically represented as angles within the range 0 to 360. This range of0 to 360 may be compressed, for example to the range 0 to 240, leaving arange of hues for representation of invisible frequencies like infrared.Embodiments using visible frequency compression provide the benefit thatdistinct colors are typically mapped to distinct colors, although thedifferences between colors may be reduced.

One or more embodiments may combine visible frequencies and invisiblefrequencies onto one or more displays. For example in an embodiment witha left eye display and a right eye display, one display may show visiblefrequencies, and the other display may show invisible frequencies (wherethese invisible frequencies are transformed into visible frequenciesusing any desired transformation function). In another embodiment,invisible frequencies may be shown in a picture-in-picture displayinserted into or beside an image showing visible frequencies. In anotherembodiment, one or more displays may alternate between displayingvisible frequencies and displaying invisible frequencies, either at afixed rate or upon receiving input from a user to change the displayedfrequency range. Any method of combining several light frequency rangesonto one or more displays is in keeping with the spirit of theinvention. With these techniques embodiments of the invention mayeffectively broaden the range of light frequencies that are visible to auser beyond the normal range of human vision. Moreover the specificfrequencies made visible to a user or emphasized in an image display maybe customized to the requirements of a particular application.

One or more embodiments map camera images into a time sequence ofdisplay images. Information of interest in the camera images maytherefore be encoded in the time varying features of the display imagesequence. For example, to assist a colorblind user in distinguishingbetween colors, different colors may be represented as pixels that flashon and off at different frequencies. Embodiments may vary the range ofcolors or the period of repetition to highlight or differentiateselected colors or other features.

One or more embodiments may add labels, captions, graphics, or otherdecorations to the display images. For example, embodiments may addlabels for selected colors to regions of those colors; these labels mayhelp colorblind users understand the color content of an image. Labelsor captions may also be used to highlight any feature of interest in acamera image.

One or more embodiments provide a user interface, or multiple userinterfaces, to support configuration or customization of the imagetransformation function. User interfaces may include physical inputdevices such as switches, buttons, or joysticks; they may includescreens with mouse, keyboard, touchscreen, stylus, or touchpad input; orthey may include voice input or any combination thereof. Screens orinput devices may be components of any device, including for example,and without limitation, a desktop computer, server computer, laptopcomputer, tablet computer, mobile phone, smart phone, pager, radio,smart watch, graphics tablet, game controller, or personal digitalassistant. In general, embodiments may use any device or combination ofdevices to accept user input. Some embodiments provide user interfacesthat provide for camera control as well, such as modifying the zoom orfield of view of one or more of the cameras of the system.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The above and other aspects, features and advantages of the inventionwill be more apparent from the following more particular descriptionthereof, presented in conjunction with the following drawings wherein:

FIG. 1 illustrates an architectural view of at least one embodiment ofthe image transforming vision enhancement device, showing elements thatinclude cameras, displays, a processor, and an image transformationfunction.

FIG. 2 illustrates a front view of an embodiment of the imagetransforming vision enhancement device that is embedded in eyeglasses.

FIG. 3 illustrates a back view of the embodiment shown in FIG. 2.

FIG. 4 illustrates an embodiment of the image transforming visionenhancement device that is embedded in a contact lens.

FIG. 4A illustrates an embodiment of the image transforming visionenhancement device where the display is embedded in a computer screen,and the device transforms images as they are sent from a server over anetwork to the computer.

FIG. 5 illustrates an embodiment of the image transforming visionenhancement device that is embedded in a monocular with a zoomcapability.

FIG. 6 illustrates an embodiment of the image transforming visionenhancement device that is embedded in eyeglasses, with a backwardfacing camera in addition to front facing cameras, and apicture-in-picture display combining the back image with the frontimages.

FIG. 7 illustrates an embodiment of the image transforming visionenhancement device that is embedded in eyeglasses, with different imagetransformation functions applied to the left eye image and the right eyeimage.

FIG. 8 illustrates an embodiment of an image transformation functionthat enhances color vision for a red-green colorblind user, by mappingcolors that such a user cannot distinguish into colors that the user caneasily distinguish.

FIG. 9 illustrates an embodiment of an image transformation functionthat highlights pixels within a designated color range.

FIG. 10 illustrates an embodiment of an image transformation functionthat reduces glare, for example in order to assist with night driving.

FIG. 10A illustrates an embodiment of an image transformation that bothreduces glare and brightens dark areas of an image, for example in orderto assist with night driving.

FIG. 11 illustrates an embodiment of an image transformation functionthat allows a user to see infrared frequencies, by mapping thesefrequencies into the visible spectrum.

FIG. 12 illustrates a different embodiment of an image transformationfunction that allows a user to see infrared frequencies, by allocating aportion of the visible hue color wheel to hue representations forinfrared hues.

FIG. 13 illustrates another embodiment of an image transformationfunction that allows a user to see infrared frequencies, by compressingvisible hues into a subset of the visible hue color wheel, and insertinginfrared hues into the remaining hue color space.

FIG. 14 illustrates an embodiment of an image transformation functionthat maps colors into a time-varying sequence of colors; in thisembodiment colors with predominantly red hues are mapped into flashingpixel values.

FIG. 15 illustrates an embodiment of an image transformation functionthat combines elements of the FIG. 8 embodiment for colorblind visionwith the FIG. 14 embodiment that provides for flashing pixels.

FIG. 16 shows an extension of the embodiment of FIG. 15, where colors indifferent ranges are mapping into pixels that flash at differentfrequencies.

FIG. 17 illustrates an embodiment of an image transformation functionthat adds a color label to regions in a specified color range.

FIG. 18 illustrates an embodiment of an image transforming visionenhancement device that provides a user interface to customize andconfigure the image transformations.

FIG. 19 illustrates another embodiment with a user interface tocustomize and configure the image transformations.

DETAILED DESCRIPTION OF THE INVENTION

An image transforming vision enhancement device will now be described.In the following exemplary description numerous specific details are setforth in order to provide a more thorough understanding of embodimentsof the invention. It will be apparent, however, to an artisan ofordinary skill that the present invention may be practiced withoutincorporating all aspects of the specific details described herein. Inother instances, specific features, quantities, or measurements wellknown to those of ordinary skill in the art have not been described indetail so as not to obscure the invention. Readers should note thatalthough examples of the invention are set forth herein, the claims, andthe full scope of any equivalents, are what define the metes and boundsof the invention.

FIG. 1 shows an architectural view of an embodiment of an imagetransforming vision enhancement device. User 100 is depicted using thedevice, although in some embodiments multiple users may use the devicesimultaneously or sequentially. For example, some embodiments mayproject images onto one or more common displays that multiple users canview. In other embodiments users may have individual displays, but mayshare common cameras so that all users can view the same image orvariants of the same image.

The embodiment shown in FIG. 1 has two cameras 121 and 122. Otherembodiments may be implemented with only one camera, or may have morethan two cameras. In this embodiment both cameras capture views of scene125, and send the captured camera images to processor 140. In someembodiments the fields of view of multiple cameras may not overlap atall, but instead they may provide vision in multiple directions. Someembodiments may use more than one processor to receive and transformimages. For example, in some embodiments a processor may be associatedwith each camera or with each display. Embodiments may use anycommunications technology to send camera images to the processor orprocessors 140, including both wired and wireless communication. Theprocessor or processors may be located near the cameras, near thedisplays, or remote from both the cameras and the displays. Theprocessor or processors may be microprocessors embedded in devices suchas glasses or binoculars, or they may be computers such as personalcomputers, laptops, tablets, mobile phones, or servers. Any computingdevice or network of computing devices capable of receiving andtransforming images may be used as a processor for the device.

Processor 140 executes an image transformation function 150 that mapscamera images into display images. The image transformation function maybe implemented in hardware, in microcode, in software, or using anycombination thereof. The image transformation function transforms imagesreceived from cameras such as 121 and 122 into images that are shown ondisplays visible to the user. In the embodiment shown in FIG. 1 thereare two displays 101 and 102. For example, displays may be placed infront of the eyes of user 100, with a separate display for each eye, asin FIG. 1 where the left eye of user 100 sees display 101 and the righteye of user 100 sees display 102. Other embodiments may utilize a singledisplay, or more than two displays. Displays may be configured to beviewed by a single eye, or by both eyes of a user simultaneously, or bymultiple users either at different times or simultaneously.

In FIG. 1 the camera image 132 is captured by camera 122. Imagetransformation function 150 maps image 132 into display image 112, whichis shown on display 102 and is viewed by the right eye of user 100.Similarly the image from camera 121 is transformed into image 111, andis shown on display 101, which is viewed by the left eye of the user.Other embodiments may have different numbers and configurations ofcameras or of displays. Any configuration or number of displays andcameras is in keeping with the spirit of the invention.

In FIG. 1, image transformation function 150 provides a colortransformation. The function maps colors of camera pixels into colors ofdisplay pixels. In some embodiments the image transformation functionmay provide transformations other than or in addition to colortransformations. For example, embodiments may include imagetransformation functions that rotate, scale, crop, translate, shift,reflect, blur, sharpen, resample, oversample, subsample, convolve,lighten, darken, or otherwise modify images in any desired manner.Images may also be combined or overlaid with one another or with othergraphics, they may be combined as picture-in-picture displays, tileddisplays, interleaved or striped displays, and they may be annotatedwith labels, icons, shapes, hyperlinks, legends, or any other desiredannotation, explanation, or decoration, or shown with time-shiftingtechniques to vary intensity over time, flash one or more colors orranges or include a pattern in an area for example.

The embodiment of image transformation function 150 maps colors incamera color space 151 into colors in display color space 152. FIG. 1shows an illustrative image transformation in which both the cameracolor space 151 and the destination color space are RGG (red-green-blue)color spaces. Any desired color space or spaces may be used for eitherof the camera color space or the display color space. Color spaces otherthan RGB may include, without limitation, RGBA, CMY, CMYK, HSV, HSL,HSI, indexed color spaces, or any convenient means of representingcolors with tuples of values. Such color spaces may or may not includealpha components representing transparency. Color spaces may alsoconsist of multidimensional vectors of color tuples, where a particularinput color tuple value (such as for example an RGB tuple) is mappedinto multiple output color tuples (such as for example a sequence of RGBtuples). These multidimensional color spaces may be used in one or moreembodiments to map input pixels into a time sequence of output pixels,where the output time dimension provides an extended dimension for theoutput color space. In FIG. 1 the illustrative image transformationfunction is defined by formula 153. This particular formula is forillustration only; embodiments may use any desired mapping defined byformulas, algorithms, software, tables, graphs, or any other means ofrepresenting functions from the source color space to the target colorspace. In the illustrative formula 153, color values in the camera colorspace are represented as xR+yG+zB, where x,y,z are values in the range 0to 255. This color space represents colors as 8 bits per channel, foreach of the 3 channels R, G, and B. Embodiments may use any desirednumber of bits for any color channel. The illustrative formula 153 mapsxR+yG+zB into yR+zG+max(x−z,0)B. FIG. 1 illustrates the effect of thismapping on the colored camera image 132, which contains a cross of white(RGB=255,255,255) on a background of red (RGB=255,0,0). White is mappedto value 255*R+255*G+(255−255)*B=255R+255G, which is yellow. Red ismapped to value 0R+0G+255B, which is blue. Thus the output 112 is ayellow cross on a blue background. This image 112 is shown on display102. A similar transformation is done on camera image from camera 121,generating display image 111, which is shown on display 101. In thisillustrative embodiment, the color transformation on the two cameraimages is identical; the display images differ only in that they capturedifferent fields of view of the target scene 125. Other embodiments mayuse different image transformations for the images from differentcameras. In addition, any mapping of color outside of visible ranges forexample may be utilized, which may for example utilize any other colorspace for example and use of RGB in this figure is exemplary only.

FIG. 2 illustrates a front view of an embodiment where the cameras anddisplays are embedded in eyeglasses. Embodiments may embed all or someof the components into any other devices, including without limitationeyeglasses, sunglasses, contact lenses, intraocular lenses, a monocular,binoculars, telescopes, microscopes, intraocular lenses, monocles,goggles, helmets, diving helmets, diving masks, welding helmets,protective eyewear, mirrors, visors, windows, windshields, televisions,game consoles, virtual reality displays or glasses or helmets, desktopcomputers, laptop computers, notebook computers, internet appliances,tablet computers, mobile phones, smart phones, personal digitalassistants, pagers, radios, smart watches, medical devices, scientificdevices, stadium monitors, or movie screens. In eyeglasses 200 of FIG.2, left camera 121 is embedded in the front of the left lens of theglasses, and right camera 122 is embedded in the front of the right lensof the glasses. The cameras are oriented to point forward and to provideimages that overlap with the normal field of view of the user. In someembodiments the camera or cameras may be oriented to capture all or aportion of the user's normal field of view. Such embodiments can provideaugmentation for the user's normal vision. In other embodiments one ormore cameras may be oriented to capture areas that are partially orcompletely outside the user's normal field of view. For example andwithout limitation, such other areas may include areas behind, above,below, or to the sides of the user, or in front of user and with a widerfield of view. Some embodiments may employ cameras that are remote fromthe user. Such embodiments may supplement the user's normal vision withadditional fields of view that the user would not normally see withoutthe vision enhancement device. One or more embodiments embedded inglasses or other eyewear may also include one or more cameras pointingat the user's eyes, and one or more displays located on the outsidesurface of the lenses where images of the user's eyes are displayed;these additional cameras and displays provide eye contact to anotherperson interacting with the user, which may otherwise be absent if theuser's eyes are occluded by the displays on the inner surface of thelenses.

FIG. 3 shows a back view of the embodiment illustrated in FIG. 2 withthe vision enhancement device embedded in eyeglasses 200. In thisembodiment the cameras 121 and 122, the displays 101 and 102, and theprocessor 140 are all embedded in eyeglasses 200. Other embodiments mayhave some or all of these components external to the eyeglasses or otherdevice. For example, in some embodiments the user may have eyeglasseswith displays at the lenses of the eyeglasses, where cameras areexternal to the eyeglasses to provide a view of a different scene. Theprocessor 140 similarly may be either internal to or external to theeyeglasses or similar device. In the embodiment shown in FIG. 3 theprocessor 140 is located along the left side of the eyeglasses frame;other embodiments may place the processor or processors in anyconvenient location. The cameras capture left and right images of scene125, transform the images using processor 140, and project thetransformed images 111 and 112 onto displays 101 and 102 respectivelythat are located on the back of the lenses of the glasses 200.

In one or more embodiments the vision enhancement device may be fully orpartially embedded in contact lenses, as illustrated in FIG. 4. Contactlens 400 includes processor 140 that processes images captured from acamera on the front of the lens, and projects the transformed image ontothe display on the back of the lens to be viewed by user 100. As withembodiments embedded in eyeglasses, some of the components may beexternal to the contact lens in one or more embodiments.

In one or more embodiments components of the vision enhancement devicemay be fully or partially embedded in a television, a monitor, a kiosk,a server computer, a laptop computer, a desktop computer, a notebookcomputer, a tablet computer, a smart phone, a smart watch, a personaldigital assistant, or in any other device that can send, transmit,receive or display images. FIG. 4A illustrates an embodiment in whichthe display of the vision enhancement device is the display of a laptopcomputer. Other elements of the vision enhancement device aredistributed on nodes that communicate via a network. In the embodimentshown in FIG. 4A, server 4A01 provides a virtual or real camera thatcaptures or stores camera images, and serves these camera images toclients over network 4A20. Images may be served for example as parts ofweb pages, video streams, other media streams, emails, or embedded inany other content delivered over a network. Network 4A20 may be theInternet, or it may be any public or private network including, withoutlimitation, a wide area network, a metropolitan area network, a localarea network, or a personal area network. Network 4A20 may consist ofwired links, wireless links, or any combination thereof. Camera image132 is stored or captured on server 4A01; it contains an image of a facewith a red color. In the illustrative embodiment shown in FIG. 4A, thedesired image transformation function 150 exchanges the red channel andthe blue channel of the image, where the image pixel values are in theRGB color space. The transformed image is displayed on client computer4A02 as display image 112. After the image transformation the displayimage 112 has a blue face. This transformation function is illustrative;embodiments may use any desired image transformation function to alterimages in any manner.

FIG. 4A illustrates three alternatives for the organization of the imagetransmission path and the location of the processor that executes imagetransformation function 150. Embodiments may use any of these threealternatives, or combinations of these alternatives. In transmissionpath 4A11, the image transformation function 150 a is performed onserver computer 4A01, or on a processor proximal to this servercomputer. The transformed image is then sent across network 4A20 toclient 4A02. The server computer may select different imagetransformation functions for different clients, based on any clientparameters or client requests. In transmission path 4A12, the imagetransformation function 150 b is performed on the client computer 4A02,or on a processor proximal to this client computer. For example, theimage transformation function may be embedded in a browser displayingweb pages, or in an email client displaying emails or email attachments.In transmission path 4A13, camera image 132 is transmitted first toproxy server 4A03, which executes image transformation function 150 c.The transformed image is then sent over network 4A20 to client 4A02. Useof any or all of these transmission paths to deliver modified images toany client is in keeping with the spirit of the invention.

FIG. 5 illustrates an embodiment of the invention embedded in monocular500. The embodiment shown has a single camera 121 on the front lens ofthe monocular, a single display 101 on the eyepiece of the monocular,and processor 140 (not shown) is installed in the interior of themonocular. Other embodiments may have cameras, displays, or processorsexternal to the monocular device. Other embodiments may embed componentsinto devices such as for example binoculars, telescopes, or microscopes.In the embodiment shown in FIG. 5, image transformation function 150provides a zoom capability, as well as a capability to increase thebrightness of dark scenes. In some embodiments a zoom capability may beprovided in the camera or cameras instead of or in addition to in theimage transformation function. In the embodiment shown, camera 121captures a portion 501 of scene 125, here an upper left portion. Thecamera image 501 is magnified 5 times by image transformation function150, and then presented as magnified image 111 on display 101. Otherembodiments may use different magnification factors, or provide a rangeof adjustable magnifications. Image transformation function 150 alsoprovides a color transformation between source and destination images.In the embodiment shown, HSL (hue-saturation-lightness) color spaces areused for camera color space 151 and for display color space 152, with H,S, and L scaled to be in the ranges [0,360), [0,1], and [0,1]respectively. The color transformation formula 153 increases thelightness of each pixel by up to 3×, by replacing lightness z withmax(3z,1). Thus dark pixels (with a small z) are mapped into brighterpixels for better visibility. Other embodiments may use differentformulas or color transformations to increase visibility of dark images.Some embodiments may employ thresholds to increase the brightness ofpixels that are below a particular value of lightness. Some embodimentsmay use color spaces other than HSL to define appropriate colortransformations, and they may use camera color spaces that differ fromdisplay color spaces. Some embodiments may attenuate the brightness ofexcessively bright images or bright image regions, instead of or inaddition to increasing the brightness of dark images or dark regions.

In one or more embodiments, the image transformation function maycombine images from multiple cameras onto a single display image. FIG. 6illustrates such an embodiment with the components of the embodimentembedded into eyeglasses 200. In this embodiment there is abackward-facing camera 601 in addition to the two front-facing camerason the front of the lenses of the glasses. Camera 601 captures an imageof scene 602 that is behind the user. Front facing cameras capture leftand right images of scene 125. The image transformation functioncombines the back image and the right image into left display image 111and right display image 112. In this embodiment, the back image is shownas a picture-in-picture insert 611 in image 111, and as apicture-in-picture insert 612 in image 112. Other embodiments maycombine images in other ways, such as for example side-by-side,top-and-bottom, blended, morphed from one image to another, tiled,striped horizontally, striped vertically, striped diagonally, oralternating in time. Any method of combining multiple images into singledisplays is in keeping with the spirit of the invention. Embodiments mayalso combine more than two camera images into a display image; forexample camera images from many different angles may be combined into asingle panoramic display image. Some embodiments of the inventiontherefore provide a complete 360° field of view for a user, or a fieldof view covering an entire sphere surrounding a user.

In one or more embodiments the image transformation function may applydifferent transformations for different displays. FIG. 7 illustrates anembodiment with different image transformation functions for the lefteye image and the right eye image. One or more cameras capture images ofscene 125. Camera images are provided to left image transformationfunction 150 a and right image transformation function 150 b. In theexample illustrated, these two image transformation functions performdifferent color mappings on the camera images. The example shown appliesdifferent color filters to the separate eyes: the left eye imagecontains only blue, and the right eye image contains only green. Thisexample is only illustrative; any transformations may be applied togenerate different display images for different displays. Use ofseparate display-specific image transformation functions may be appliedfor example to generate 3D vision using standard anaglyph techniques. Inother embodiments image transformation functions themselves may generate3D stereoscopic vision by offsetting pixels in left and right imagesusing an auxiliary depth map as input to the transformations. Additionalapplications may include for example correction of vision problems suchas cross-eyed vision or heterotropia or strabismus, where a user's eyesare not aimed directly forward.

Embodiments of the invention may be used to correct for visiondeficiencies such as full or partial colorblindness. FIG. 8 illustratesan embodiment that may assist a red-green colorblind user. Such a usermay find it difficult or impossible to distinguish between certain redand green pixel values. For simplicity, FIG. 8 illustrates an examplewhere a colorblind user sees only green, and where this user also seesred pixel values as green. As shown, square 801 a has a dark green colorwith RGB value (0,128,0) (with R, G, and B each in the range 0 to 255).Square 802 a has a dark red color with RGB value (128,0,0). A normaluser can easily distinguish these two colors. However the red-greencolorblind user sees both squares as dark green: 801 a is seen as 801 b,with RGB of (0,128,0), and 802 a is seen as 802 b, with RGB also of(0,128,0). The colorblind user therefore cannot distinguish thesecolors.

In the embodiment shown in FIG. 8, image transformation function 150separates the colors of 801 a and 802 a so that the red-green colorblinduser can see a difference. In this illustrative example, the R, G, and Bvalues are scaled up or down depending on the difference between red andgreen. Pixels with more red than green are increased in RGB intensity;pixels with more green than red are decreased in RGB intensity. Imagetransformation function 150 maps square 801 a into 801 c, with RBG value(0,64,0); this square is made darker since its green value exceeds itsred value. Square 802 a is mapped into 802 c, with RGB value (192,0,0);this square is made lighter since its red value exceeds its green value.The red-green colorblind user sees these transformed images as 801 d and802 d, with RGB colors of (0,64,0) and (0,192,0) respectively. Thecolorblind user can then distinguish these colors based on theirrelative brightness. The transformation function illustrated in FIG. 8is only an example of the possible transformations that can assist withcolorblind vision. Appropriate transformations may depend on thespecific type and degree of colorblindness of a user, as well as on thecolors in the original image. Some embodiments may provide options forusers to select appropriate transformations based on their specificvision issues. Other embodiments may be designed to be appropriate forparticular vision types. Any transformation function or functions thatassist users in distinguishing colors is in keeping with the spirit ofthe invention, including any time-shifting, flashing, pattern overlayingor time-varying color mapping for example or any combination thereof.

One or more embodiments of the invention may use an image transformationfunction that first applies an image mask to select particular pixelsfrom a camera image, and then applies a modification to the selectedpixels. An image mask may be adaptive in that it is defined by thecharacteristics of the pixels themselves, rather than by a fixedgeometric boundary. Fixed boundary image masks may also be used insteadof or in addition to adaptive image masks. As an example, one or moreembodiments may use image masks that select pixels having colors in aspecified color range or a set of color ranges. Modifications to thepixels selected by a color-based mask may for example be used tohighlight the pixels in the specific color ranges. FIG. 9 illustrates anembodiment with an image transformation function 150 that highlightspixels with RGB values the range of (67,92,127)-(127,152,187), which isroughly a medium blue color. This range of RGB values defines anadaptive image mask based on color. For pixels in this range, thefunction doubles the intensity of each of the RGB values, up to themaximum value of 255. This transformation has the effect of brighteningand therefore highlighting these pixels. Thus camera image 132 istransformed by function 150 into display image 112. Pixel 901, as anexample, is brightened and highlighted by this transformation. Otherembodiments may use different transformations to highlight pixels inspecific color ranges. For example, one embodiment might transformpixels outside the selected range into grayscale pixels, whilepreserving the color of pixels in the designated range. Anotherembodiment may add graphics such as an outline or a pointer to regionsin a specified color range or vary any viewing parameter over time forexample. Any transformation that highlights the desired pixels is inkeeping with the spirit of the invention. Some embodiments may provideuser-selectable or user-configurable color ranges to be highlighted. Apotential application of such embodiments might include for exampleinspection devices that assist inspectors in identifying specific partsor regions or articles that have known color signatures.

One or more embodiments may alter the brightness of pixels or regions toimprove visibility. As illustrated in FIG. 5, embodiments may lightendark pixels to provide improved visibility of dark scenes. FIG. 10illustrates the converse requirement of darkening overly bright pixels.Such embodiments may have applications for example in nighttime drivingwhere bright headlights can be hazardous, or in industrial applicationssuch as welding where bright emissions can be damaging. In FIG. 10camera image 132 contains an excessively bright region 1001corresponding to the bright headlights of a car. Image transformationfunction 150 attenuates the brightness of very bright pixels. In thisillustrative example, pixels with R, G, and B values all above 220 havetheir intensity reduced by half. The resulting image 112 has darkenedpixels in the previously bright areas, such as darkened area 1002 forthe headlight. FIG. 10A illustrates an embodiment of the invention thatsimultaneously darkens excessively bright regions, and brightensexcessively dark regions. This embodiment or similar embodiments provideadditional advantages for example in nighttime driving applicationswhere there is a need to simultaneously improve low-light visibility andreduce excessive glare. In the embodiment shown in FIG. 10A, as in theembodiment illustrated in FIG. 10, camera image 132 contains anexcessively bright region 1001 corresponding to the bright headlights ofa car. Moreover the image of the car 10A01 is very dark, making itdifficult to distinguish the car from the background. Imagetransformation 10A02 illustrates an image transformation defined by acurve; in general embodiments may specify image transformations usingany combination of techniques including, without limitation, tables,formulae, charts, graphs, curves, algorithms, software, databases,expression languages, or any other mechanism for specifying atransformation from inputs to outputs. In this example curve 10A02defines a mapping from HSV (hue-saturation-value) space into HSV space;the illustrative transformation modifies only the value component (V-in)of each input pixel to generate a value component (V-out) of the outputpixel. In other embodiments transformations may alter any or all of thechannels of an image to modify brightness, colors, contrast, hues,saturation, luminance, or any other image characteristics in any desiredmanner. Transformations may use any desired color space or combinationof color spaces; the HSV space used here is illustrative. Thetransformation defined by curve 10A02 amplifies the value of dark pixels(with a small V-in), and it reduces the value of bright pixels (with alarge V-in). This illustrative transformation has the effect of reducingexcessive glare while simultaneously boosting visibility of darkregions. The resulting display image 112 has darkened area 1002 for theoverly-bright headlight. It also has a clearer, brighter image 10A03 forthe car, which stands out more clearly against the dark background afterapplying image transformation 150.

In some embodiments devices with such brightness-altering imagetransformations may be embedded for example in glasses, sunglasses,contact lenses, visors, helmets, goggles, protective eyewear, or otherwearable devices. The portion of the image to transform may include anyalpha value and any pattern, here shown as a monochrome grey pattern forease of illustration. One skilled in the area will recognize that thegrey pixel pattern may replaced with a hash or checkerboard or any otherpattern. In other embodiments they may be embedded directly in movingvehicles such as cars or trucks, for example within windshields,rear-view mirrors, visors, or any other component of the vehicle. Someembodiments may provide daytime and nighttime modes, or other modes thatare time or condition specific, that alter the image transformationfunction to enhance vision appropriately in the different environments.

One or more embodiments of the invention may include one or more camerasthat can capture wavelengths outside of the visible range of light. Suchwavelengths may for example include infrared, ultraviolet, gamma waves,x-rays, microwaves, radio waves, or any other part of theelectromagnetic spectrum. Any sensor that can capture a range or any setof ranges of the electromagnetic spectrum may be used as one or morecameras in embodiments of the invention. To make such wavelengthsvisible to a user, some embodiments may employ image transformationfunctions that map these wavelengths into the visible spectrum. FIG. 11illustrates such an embodiment with a camera that captures infraredlight as well as visible light. The spectrum 1101 visible to the cameraranges from blue to infrared. For illustration, FIG. 11 presumes thatthe camera generates pixels with 4 color channels, corresponding to blueat a wavelength of roughly 450 nm, green at roughly 525 nm, red atroughly 700 nm, and infrared at roughly 900 nm. Spectrum 1102 is asubset of 1101 corresponding to visible light; spectrum 1103 is a subsetof 1101 corresponding to invisible light (which in this case isinfrared). Other embodiments may use other number of color channels andmay have color channels corresponding to different wavelengths; thesevalues are for illustration only. In the embodiment of FIG. 11, imagetransformation function 150 maps the entire spectrum 1101 onto thevisible spectrum 1102. The color space 151 corresponding to the entirespectrum 1101 is the 4-channel space RGB X IR, and the color space 152corresponding to the visible spectrum 1102 is the 3-channel space RGB.In this example, this mapping is performed by mapping the visiblespectrum 1102 onto a smaller visible spectrum 1104, which includesroughly blue and green but not red, and by mapping the invisiblespectrum 1103 onto a different portion 1105 of the visible spectrum,which corresponds roughly to red. Function 150 illustrates one simplemethod for this mapping: in this example, the red channel is discarded,and the infrared channel is mapped onto the red channel. Otherembodiments may use other techniques to map the entire spectrum onto thevisible spectrum, such as discarding, combining, compressing, ormodifying any or all of the channels. Mapping from discrete colors in anon-linear manner to provide false color or random mappings may beemployed in one or more embodiments of the invention and range to rangemapping is illustrated in FIG. 11 as an example of one type of mappingthat embodiments of the invention may utilize.

FIG. 12 illustrates an embodiment of a different image transformationfunction that maps a spectrum including invisible frequencies into thevisible spectrum. In this embodiment visible pixels are represented inHSV (hue-saturation-value) color space rather than in RGB. Color wheel1102 is simplified version of a standard hue wheel for the hue componentof this color space. In the embodiment shown, the camera color space,which includes infrared frequencies, expands the usual visible hue spacewith an additional range 1103 of hues between red and infrared. Theseinfrared hues do not correspond to visible colors, but they can bedefined and calculated easily by measuring for example the relativemagnitude of red and infrared channels in a pixel. For illustration thisrange 1103 is shown as comprising a 120° range. In this embodiment aportion 1104 of the visible hue range 1102 is maintained unchanged bythe image transformation function; the remaining portion of the visiblehue range 1102 is allocated for infrared hues. Thus infrared hues inthis example are mapped into visible hues 1105 between red and blue inthe color wheel.

Embodiments like the one illustrated in FIG. 12 provide simple mappingsof invisible frequencies into the visible spectrum, but they lose somecolor information in the mapping. For example, in the embodiment of FIG.12, two colors with identical red and infrared values, but differentblue values, could be mapped into identical colors. Some embodimentspreserve more color differences by compressing the visible spectrum in aone-to-one manner. FIG. 13 illustrates such an embodiment. As in FIG.12, in this embodiment colors are represented in HSV color space, withextended hues 1103 defined between red and infrared. The visible hues1102 are mapped into compressed hue space 1104 by function 150. Thisillustrative mapping scales hue values by the factor 2/3, so that the360° visible hue space is compressed onto the compressed space 1104,which comprises 240° of the original space. This allocates the remaining120° for the infrared hues 1103. In contrast to the embodiment of FIG.12, the embodiment of FIG. 13 preserves color distinctions becausedifferent hues in the camera color space are mapped into different huesin the display color space. Differences are compressed in thisembodiment but they are not eliminated. Other embodiments may employdifferent compression strategies to map colors onto compressed colorspaces. For example, some embodiments may preserve all hue valuesunchanged within a particular hue range, and compress others into acompressed range. Combinations of techniques may be used to createvisible representations of the invisible frequencies, while preservingdesired aspects of the visible frequencies that are relevant forparticular applications.

In one or more embodiments of the invention, one or more sensors ofphysical signals other than electromagnetic radiation may be used ascameras. These sensors may be used in addition to sensors detectingelectromagnetic radiation, or they may be used without sensors ofelectromagnetic radiation. Such sensors may include for example, withoutlimitation, microphones, sonar, Geiger counters (which may also detectradiation such as electromagnetic radiation), inertial sensors, forcesensors, pressure sensors, vibration sensors, temperature sensors,strain gauges, or flow sensors. In embodiments with sensors ofnon-electromagnetic signals, the camera images consist of sensormeasurements over time for a set of pixels that represent locations orother points of interest in the object or object being observed. Thesemeasurements may be single channel values or multi-channel values. Themeasurements may be made on any desired scale and within any desiredrange. The image transformation function in these embodiments maps thesensor values into display images using any desired function. Forexample, one embodiment may measure sound levels at various points andmay display the sound field as a display image, with different colorscorresponding to different decibel levels of sound. The mapping fromsound level, or any other physical signal, into display images may beselected as desired for the specific application. One or moreembodiments may employ sensor fusion techniques to integrate data frommultiple sensors into one or more display images. The imagetransformation function in embodiments using sensor fusion maps tuplesof sensor readings from multiple sensors into display images.

One or more embodiments may display information on sounds in theenvironment for example to assist deaf or hearing-impaired users, or toaugment the hearing of users in noisy environments, or in otherenvironments where normal hearing is difficult. For example, one or moreembodiments may identify specific sounds or specific sources of soundsand may display information on these sounds on one or more displays ofthe embodiment. In an illustrative embodiment, for example, noisesidentifying particular hazards, such as sirens, car horns, or alarmbells, may be recognized by the embodiment and may trigger a display ofa visual indicator of the hazard, such as for example a warning icon ora flashing color. As an example, an embodiment that normally showscamera images of the environment may have a special graphic or overlayshown when a hazardous noise is detected. The graphic may furtherindicate the type of noise and the intensity of the sound. In one ormore embodiments, the information displayed about a sound may includeindications of the direction and distance to the source of the sound.Such displays may for example include arrows and range informationpointing towards the source, and they may include a map showing thelocation of the source. As an illustration, an embodiment that displaysthe direction of nearby sounds above a threshold or matching a patternmay for example alert hearing-impaired pedestrians of oncoming traffic.Sounds of interest may be determined by loudness, pitch, patterns, orany other characteristics of the relevant sounds. One or moreembodiments may generate graphical overlays or labels to annotate imagesof objects or people in the environment based on the sounds they aregenerating. For example, an embodiment may use graphic overlays tohighlight images of people in a camera image that are currently talking.One or more embodiments may use speech recognition technology totranslate spoken words into text or into other symbols that are shown onthe displays of the embodiment. One or more embodiments may detectsounds outside the normal range of human hearing, and may displayinformation about these inaudible sounds on the displays of theembodiment.

In one or more embodiments of the invention, image transformationfunctions may map camera images to a time sequence of display images.The time-varying characteristics of individual pixels or regions maytherefore provide visual cues to users. This may include flashing acolor or ranges of colors, changing the color over time to any othercolor or sequence of colors including a linear range, non-linear range,random mapping or any other mapping for example and may include anypattern overlay as well. In these embodiments the display color spacemay be viewed as a composition of a standard color space with a timeaxis. The time range for the time sequence of display images may belimited or unlimited. For embodiments that map a camera image to afinite sequence of display images, the sequence of display images may berepeated in some embodiments after the last display image is reached. Inother embodiments the sequence of images may not repeat, but instead thelast display image or the first image or any other image may be shownafter reaching the end of the sequence. In some embodiments the cameraor cameras may capture new camera images continuously.

One or more embodiments may generate a time sequence of display imagesthat alternates or otherwise interleaves display of values in differentfrequency ranges. For example, embodiments may display the visiblefrequencies from a camera image for some period of time, and then switchto a display of the invisible frequencies of the camera image (such asultraviolet or infrared or other frequencies) for another period oftime. This time multiplexing of different frequency channels onto thedisplay image may be used instead of or in addition to the frequencyspace mapping and compression techniques described above, that fitinvisible frequencies into the visible spectrum.

To apply an image transformation that generates a sequence of displayimages from each camera image, some embodiments may down-sample thesequence of incoming camera images. For example, if an embodimentgenerates a time sequence of 10 display images for each camera image,and if a camera captures 30 frames per second, an embodiment may useevery 10^(th) frame from the camera and generate an output for displayat 30 frames per second using the image transformation function tointerpolate between the resampled camera frames. Any method ofdisplaying time-varying images is in keeping with the spirit of theinvention.

FIG. 14 illustrates an embodiment of the invention that maps cameracolors onto a time-varying sequence of colors in the display colorspace. In this example, the camera color space 151 is an HSV space, andthe display color space 152 is HSV X Time, with time extendingindefinitely until a new image arrives. Pixels in the red portion of thehue color wheel are mapped into alternating red pixels 1401 and whitepixels 1402; other hues are unchanged. The flashing of red pixels servesto highlight those pixels for the user. Colors may alternate at anydesired frequency, including at frequencies that vary in time. In someembodiments color mappings may map certain colors to continuouslyvarying target colors over time. Some embodiments may use time-varyingcolors or pixels to highlight certain regions or figures or shapes, toprovide flashing or scrolling captions or annotations, to provide movinggraphics, to dynamically rotate, scale, blur, or otherwise modify theimage, or more generally to provide any desired display of time-varyinginformation based on the camera image input to the image transformationfunction.

Some embodiments may use image transformation functions that generate atime sequence of output images to assist vision-impaired users. Forexample, FIG. 15 illustrates an embodiment that uses a time-varyingimage output to help colorblind users distinguish between colors thatthey are normally unable to distinguish. As in the example shown in FIG.8, the example in FIG. 15 presumes a red-green colorblind user thatcannot differentiate between red and green hues. Camera image 132 haseyes with a dark red color with RGB value (128,0,0), and has a face witha dark green color (0,128,0); the red-green colorblind user sees thesecolors as identical. Image transformation function 150 maps the dark redcolor to a time sequence of alternating dark red and white colors. Inthis example the colors alternate every second. Embodiments may use anydesired frequency for varying or alternating colors, and may apply colortransformations to any subset of colors or to all colors as required forthe application. The display image output 112 from function 150 is asequence of display images that change every second. For example, thefirst frame 1501 has dark red eyes 1503. The second frame 1502 is shownafter one second has elapsed; it has white eyes 1504. The red-greencolorblind user can see the difference between the white eyes 1504 andthe dark green background of the face, although he or she cannot see thedifference between the eyes and the background in frame 1501.

One or more embodiments may generate time-varying outputs that vary atdifferent frequencies, where the frequency depends on the input color oron any other characteristics of the camera image. This technique mayfurther assist users in identifying or highlighting specific colors.FIG. 16 illustrates and embodiment that applies a time-varying transformto all pixels with a predominantly red hue. In this embodiment function150 maps pixels with predominantly dark red color (r<=128) into colorsthat alternate to white every 3 seconds, and it maps pixels withpredominantly bright red color (r>128) into colors that alternate toblue every 1 second. This illustrative embodiment therefore generatesdifferent colors and different frequencies of color alternationdepending on the color of the input. In display image sequence 112 thedark red eyes 1503 switch to white color after 3 seconds in frame 1602.The bright red mouth 1601 switches more rapidly to a blue color at frame1502. Users can therefore use both the frequency of flashing and thedifferences in color to distinguish or highlight different shades of redin this example. Embodiments may employ any desired mixture of frequencyvariation, color variation, pattern overlay or variation, orcombinations thereof to transform images in any desired manner.

In one or more embodiments, image transformation functions may addlabels, annotations, graphics, or any other decoration to images. Forexample, labels or outlines or graphics may be added to regions withspecific colors. Such labels may include an identifying name of thecolor or color range, or any other information that assists a user inrecognizing or highlighting a color. Some embodiments may add labels orgraphics to pixels or pixel regions with any other features of interest,such as particular shapes, textures, sizes, orientations, outlines,hues, brightness, consistency, or any other recognizable characteristic.Addition of labels or graphics may be done in addition to any othertransformations such as for example modification of colors or generationof time-varying display images. FIG. 17 illustrates an embodiment withimage transformation function 150 that adds a color label to regionswith a predominantly yellow hue. In this example, function 150 uses anHSV color representation for pixels of camera image 132. It scans theimage 132 for contiguous regions where all pixels in the region have ahue value between 50 and 70 (where pure yellow has a hue of 60). In thisembodiment the image transformation function scans for regions of atleast 30 pixels by 30 pixels, and labels them with “Yellow.” Cameraimage 132 has only one such region, which is labeled at 1701 in displayimage 112. Region 1702 has the correct hue, but it is too small andtherefore is not labeled. Embodiments may use any combination offeatures to select labels or graphics. In some embodiments color labelsor graphics may be added to assist colorblind users in identifyingcolors that they are otherwise unable to distinguish. Other embodimentsmay use labels or graphics to highlight regions or pixels for anydesired application, such as highlighting regions with specified colorsfor inspection applications for example.

One or more embodiments of the invention may incorporate one or moreuser interfaces that users can employ to configure or customize thevision enhancement device. User interfaces may consist of screens,buttons, levers, knobs, switches, voice inputs, gesture recognition,mice or trackballs, touchpads or touchscreens, or any other device ordevices that accept input from a user. User interfaces may be standalonedevices or hardware, or they may be embedded in any other deviceincluding for example, without limitation, desktop computers, laptopcomputers, notebook computers, tablet computers, gaming systems, smartphones, smart watches, or personal digital assistants. User interfacedevices may be integrated into the device or devices containing thesystem's displays, cameras, or processors, or they may be separatedevices that communicate with the system. Any device that cancommunicate or exchange information with the system's components may beused as a user interface device. Some embodiments may provide multipleuser interfaces, each of which can be used to configure or customize allor part of the operation of the vision enhancement device. FIG. 18illustrates an example of a user interface for an embodiment thatprovides a computer-based user input screen. This example user interfaceprovides options to create and edit color transformation rules, and tomanage pan and zoom features for selecting and magnifying images. Theseoptions are illustrative examples; embodiments may provide any desireduser interface to modify any aspect of the operation of the visionenhancement device. In FIG. 18, existing Color Mapping Rules 1800 arepresented in a list. These rules are used to generate imagetransformation function 150 shown in FIG. 1. In this example there arethree rules currently defined at 1801, 1802 and 1803. Each rulespecifies color criteria for selection of pixels, and transformation tothe color of the selected pixels. The user can select a rule and edit itusing the features in Mapping Rule Editor 1810. The user can also deletean existing rule using button 1804, or clear all existing rules. In theexample shown the user has selected rule 1802 and the details for thisrule are presented in 1810. The user selects or modifies a color spacefor the rule; in this example the rule uses color space RGB at 1811. Therule editor provides color sliders 1812 to select the ranges of colorsto which the rule applies. The color mapping is specified with acombination of color sliders and direct input in text boxes. In thisexample the selected pixels have red maximized to 255 in input box 1813,and the blue value is reduced by half in input box 1814. Input box 1814contains an expression “b/2” rather than a specific numerical value, toindicate that the output color depends on the value of the input color.Embodiments may use any desired expression language or graphicallanguage to define color mappings or other transformations. In additionto sliders and input boxes, such inputs may include, without limitation,graphs, tables, charts, standard programming languages, specializeddomain-specific programming languages or expression languages,databases, XML definitions, dataflow diagrams, or any other techniquefor specifying rules for transformations. In the example of FIG. 18 theuser can modify the selected rule as desired and save the changes usingbutton 1815. Although shown with three color space settings, any numberof color spaces may be implemented, for example that include frequencyranges typically not visible to the naked eye.

The embodiment illustrated in FIG. 18 also includes a user interfaceelement 1812 for a Camera Manager. This interface provides zoom and pancapabilities. These capabilities may be implement in the camerasthemselves, in the image transformation function, or using a combinationof camera features and image transformation function features. In thisexample the user can select a zoom level using slider 1821. For amagnified view, the user can also select the field of view using the Panscreen; in this example the user can drag the view box 1822 to selectdifferent areas to display and magnify. Some embodiments may providephysical controls for zoom or pan controls, such as joysticks, buttons,knobs, or other input devices. Embodiments may also provide user inputsfor other image transformation capabilities, such as for examplerotating images or sharpening images.

FIG. 19 illustrates another example of a user interface for anembodiment that provides a computer-based user input screen. Thisembodiment illustrates a different organization for the controls of theimage transformation function. Embodiments may use any desiredorganization for user interface controls. The user interface shown inFIG. 19 includes a live display of camera image 132 and display image112. These image displays provide immediate feedback to the user on theeffects of modifications to the image transformation function. Colortransformations are configured in RGBA transform matrix 1901 and in HSVcontrols 1902. In this embodiment image transformations may be specifiedusing either color space RGBA or color space HSV. Embodiments may useany color space or any combination of color spaces to define andconfigure image transformation functions. Pull-down menu 1905 selectsthe layer to which the image transformation definitions apply. In thisembodiment different image transformations may be created in multiplelayers and the layers can then be intermixed or merged via blending,overlaying (using alpha channels for transparency), masking, or wipingor flashing to alternate layers over time. The user interface shown inFIG. 19 also includes timing control 1903 to control flashing of frames,and mask control 1904 to define and edit image masks that select pixelsfor which the other transformations apply.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

What is claimed is:
 1. An image transforming vision enhancement devicecomprising: one or more displays that present display images to one orboth eyes of a user, wherein each of said display images comprises aplurality of pixels with values in a display color space; one or morecameras configured to capture camera images, wherein each of said cameraimages comprises a plurality of pixels with values in a camera colorspace; a processor coupled to said one or more displays and to said oneor more cameras, wherein said processor receives said camera images fromsaid one or more cameras; transforms said camera images into saiddisplay images via an image transformation function; sends said displayimages to said one or more displays to be viewed by said user; whereinsaid image transformation function maps values from said camera colorspace into values of said display color space.
 2. The image transformingvision enhancement device of claim 1, wherein said one or more displaysare configured to be placed or worn proximal to said one or both eyes ofsaid user; said one or more cameras are proximal to said one or moredisplays; and the field of view of said one or more cameras has anon-empty intersection with the normal field of view of said user. 3.The image transforming vision enhancement device of claim 1, whereinsaid one or more displays, or said one or more cameras, or both of saidone or more displays and said one or more cameras, are located withineyeglasses or contact lenses or intraocular lenses, or within atelevision or a game system or a desktop computer or a laptop computeror a tablet computer or a mobile phone or a personal digital assistantor a diving helmet or a welding helmet or a medical device or ascientific device, or are coupled to a moving vehicle.
 4. The imagetransforming vision enhancement device of claim 1, wherein said one ormore displays are coupled to a local area network or Internet; saidcamera images, or said display images, or both of said camera images andsaid display images are transmitted over said network to said one ormore displays.
 5. The image transforming vision enhancement device ofclaim 1, wherein said one or more cameras, said image transformationfunction, or both, provide a zoom capability to increase or reducemagnification of said display images relative to said user's normalvision.
 6. The image transforming vision enhancement device of claim 5,further comprising a user interface that accepts input from an operator,wherein said input is used to select or modify the magnification, or thefield of view, or both the magnification and the field of view, of saidone or more cameras or of said image transformation function.
 7. Theimage transforming vision enhancement device of claim 5, wherein saidone or more displays, or said one or more cameras, or both of said oneor more displays and said one or more cameras, are located withinbinoculars or a monocular or a telescope or a microscope.
 8. The imagetransforming vision enhancement device of claim 1, further comprising aplurality of said cameras, wherein said image transformation functioncombines said camera images from two or more of said cameras onto asingle display image.
 9. The image transforming vision enhancementdevice of claim 8 wherein said image transforming function combines saidcamera images from two or more of said cameras into a picture-in-picturedisplay image.
 10. The image transforming vision enhancement device ofclaim 8, wherein the field of view of at least one of said cameras has anon-empty intersection with the normal field of view of said user; andthe field of view of at least one other of said cameras has an emptyintersection with the normal field of view of said user.
 11. The imagetransforming vision enhancement device of claim 1, further comprising aplurality of said displays, wherein said image transformation functioncomprises a plurality of display-specific image transformationfunctions; each of said display-specific image transformation functionsgenerates said display images for its corresponding display.
 12. Theimage transforming vision enhancement device of claim 11 wherein two ormore of said display-specific image transformation functions map atleast one value from said camera color space into different values insaid display color space.
 13. The image transforming vision enhancementdevice of claim 1, wherein said image transformation function maps twoor more values in said camera color space that are difficult orimpossible for a vision-impaired user to distinguish into values in saiddisplay color space that said vision-impaired user can distinguish. 14.The image transforming vision enhancement device of claim 1, whereinsaid image transformation function comprises: an image mask that selectspixels in said one or more camera images having said color values in oneor more color ranges of interest; and a selected pixel modificationfunction that modifies said pixels in said one or more camera imagesthat are selected by said image mask.
 15. The image transforming visionenhancement device of claim 1, wherein said image transformationfunction alters the brightness of one or more of said pixels in saidcamera images to improve visibility of dark scenes, or to reduce thebrightness of excessively bright regions, or both.
 16. The imagetransforming vision enhancement device of claim 1, wherein said cameracolor space includes one or more light frequencies outside of the humanvisible range of light.
 17. The image transforming vision enhancementdevice of claim 16, wherein said display color space representsfrequencies or frequency mixtures within the human visible range oflight; said image transformation function partitions said display colorspace into a visible frequency representation subspace and an invisiblefrequency representation subspace; said image transformation functionmaps values from said camera color space with one or more frequenciesoutside of the human visible range of light into said invisiblefrequency representation subspace; said image transformation functionmaps values from said camera color space that represent frequencies orfrequency mixtures within the human visible range of light into saidvisible frequency representation subspace.
 18. The image transformingvision enhancement device of claim 17, wherein said image transformationfunction maps distinct values in said camera color space that representfrequencies or frequency mixtures within the human visible range oflight into distinct values in said visible frequency representationsubspace.
 19. The image transforming vision enhancement device of claim18, wherein said camera color space comprises a camera hue value foreach of said pixels of said camera images; said display color spacecomprises a display hue value for each of said pixels of said displayimages, wherein said display hue values are within a display hue range;said image transformation function partitions said display hue rangeinto a compressed visible hue range and an invisible hue representationrange; maps distinct camera hue values in said camera color space intodistinct display hue values in said display hue range; maps camera huevalues in said camera color space that represent frequencies orfrequency mixtures within the human visible range of light into displayhue values in said compressed visible hue range; and, maps camera huevalues in said camera color space that contain frequencies outside thehuman visible range of light into display hue values in said invisiblehue representation range.
 20. The image transforming vision enhancementdevice of claim 1, wherein said display color space comprises a timeseries of color values for each of said pixels of each of said displayimages; said processor receives said camera images from said one or morecameras; transforms each of said camera images into a time series ofsaid display images via said image transformation function; sends saidtime series of said display images to said one or more displays to bedisplayed in sequence and viewed by said user; and wherein said imagetransformation function maps values from said camera color space intosaid time series of color values of said display color space.
 21. Theimage transforming vision enhancement device of claim 20, wherein saidimage transformation function maps two or more values in said cameracolor space that are difficult or impossible for a vision-impaired userto distinguish into different time series values in said display colorspace that said vision-impaired user can distinguish.
 22. The imagetransforming vision enhancement device of claim 21, wherein saiddifferent times series values comprise repeating time series values withdifferent periods.
 23. The image transforming vision enhancement deviceof claim 1, wherein said image transformation function further addslabels or graphics or patterns onto said display images that identifyone or more colors.
 24. The image transforming vision enhancement deviceof claim 1, further comprising a user interface that accepts input froman operator, wherein said input is used to select or modify said imagetransformation function.
 25. The image transforming vision enhancementdevice of claim 1, further comprising a microphone coupled with saidprocessor wherein said processor generates display information relatedto sound obtained from said microphone and sends said displayinformation related to sound to said one or more displays to be viewedby said user.